Distributing an executable job load file to compute nodes in a parallel computer

ABSTRACT

Distributing an executable job load file to compute nodes in a parallel computer, the parallel computer comprising a plurality of compute nodes, including: determining, by a compute node in the parallel computer, whether the compute node is participating in a job; determining, by the compute node in the parallel computer, whether a descendant compute node is participating in the job; responsive to determining that the compute node is participating in the job or that the descendant compute node is participating in the job, communicating, by the compute node to a parent compute node, an identification of a data communications link over which the compute node receives data from the parent compute node; constructing a class route for the job, wherein the class route identifies all compute nodes participating in the job; and broadcasting the executable load file for the job along the class route for the job.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with Government support under Contract No.0A-45527 awarded by the Department of Energy. The Government has certainrights in this invention.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The field of the invention is data processing, or, more specifically,methods, apparatuses, and products for distributing an executable jobload file to compute nodes in a parallel computer that includes aplurality of compute nodes coupled for data communications over a datacommunications network.

2. Description of Related Art

The development of the EDVAC computer system of 1948 is often cited asthe beginning of the computer era. Since that time, computer systemshave evolved into extremely complicated devices. Today's computers aremuch more sophisticated than early systems such as the EDVAC. Computersystems typically include a combination of hardware and softwarecomponents, application programs, operating systems, processors, buses,memory, input/output devices, and so on. As advances in semiconductorprocessing and computer architecture push the performance of thecomputer higher and higher, more sophisticated computer software hasevolved to take advantage of the higher performance of the hardware,resulting in computer systems today that are much more powerful thanjust a few years ago.

Modern computing systems can include parallel computers that include aplurality of independent compute nodes. Each of the independent computenodes may execute one or more jobs. In order to execute the one of morejobs, however, it may be necessary to load a particular compute nodewith computer code and data that is needed to execute the job.

SUMMARY OF THE INVENTION

Methods, apparatuses, and products for distributing an executable jobload file to compute nodes in a parallel computer, the parallel computercomprising a plurality of compute nodes coupled for data communicationsover a data communications network, including: determining, by a computenode in the parallel computer, whether the compute node is participatingin a job; determining, by the compute node in the parallel computer,whether a descendant compute node is participating in the job;responsive to determining that the compute node is participating in thejob or that the descendant compute node is participating in the job,communicating, by the compute node to a parent compute node, anidentification of a data communications link over which the compute nodereceives data from the parent compute node; constructing a class routefor the job, wherein the class route identifies all compute nodesparticipating in the job; and broadcasting the executable load file forthe job along the class route for the job.

The foregoing and other objects, features and advantages of theinvention will be apparent from the following more particulardescriptions of example embodiments of the invention as illustrated inthe accompanying drawings wherein like reference numbers generallyrepresent like parts of example embodiments of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example system for distributing an executable jobload file to compute nodes in a parallel computer according toembodiments of the present invention.

FIG. 2 sets forth a block diagram of an example compute node useful in aparallel computer capable of distributing an executable job load file tocompute nodes according to embodiments of the present invention.

FIG. 3 sets forth a block diagram of an example Point-To-Point Adapteruseful in systems for distributing an executable job load file tocompute nodes in a parallel computer according to embodiments of thepresent invention.

FIG. 4 sets forth a block diagram of an example Global Combining NetworkAdapter useful in systems for distributing an executable job load fileto compute nodes in a parallel computer according to embodiments of thepresent invention.

FIG. 5 sets forth a line drawing illustrating an example datacommunications network optimized for point-to-point operations useful insystems capable of distributing an executable job load file to computenodes in a parallel computer according to embodiments of the presentinvention.

FIG. 6 sets forth a line drawing illustrating an example globalcombining network useful in systems capable of distributing anexecutable job load file to compute nodes in a parallel computeraccording to embodiments of the present invention.

FIG. 7 sets forth a flow chart illustrating an example method fordistributing an executable job load file to compute nodes in a parallelcomputer according to embodiments of the present invention.

FIG. 8 sets forth an example of a parallel computer where the computenodes in the parallel computer are logically organized into a binarytree according to embodiments of the present invention.

FIG. 9 sets forth a flow chart illustrating an additional example methodfor distributing an executable job load file to compute nodes in aparallel computer according to embodiments of the present invention.

FIG. 10 sets forth a flow chart illustrating an additional examplemethod for distributing an executable job load file to compute nodes ina parallel computer according to embodiments of the present invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Example methods, apparatuses, and products for distributing anexecutable job load file to compute nodes in a parallel computer inaccordance with the present invention are described with reference tothe accompanying drawings, beginning with FIG. 1. FIG. 1 illustrates anexample system for distributing an executable job load file to computenodes in a parallel computer according to embodiments of the presentinvention. The system of FIG. 1 includes a parallel computer (100),non-volatile memory for the computer in the form of a data storagedevice (118), an output device for the computer in the form of a printer(120), and an input/output device for the computer in the form of acomputer terminal (122).

The parallel computer (100) in the example of FIG. 1 includes aplurality of compute nodes (102). The compute nodes (102) are coupledfor data communications by several independent data communicationsnetworks including a high speed Ethernet network (174), a Joint TestAction Group (‘JTAG’) network (104), a global combining network (106)which is optimized for collective operations using a binary tree networktopology, and a point-to-point network (108), which is optimized forpoint-to-point operations using a torus network topology. The globalcombining network (106) is a data communications network that includesdata communications links connected to the compute nodes (102) so as toorganize the compute nodes (102) as a binary tree. Each datacommunications network is implemented with data communications linksamong the compute nodes (102). The data communications links providedata communications for parallel operations among the compute nodes(102) of the parallel computer (100).

The compute nodes (102) of the parallel computer (100) are organizedinto at least one operational group (132) of compute nodes forcollective parallel operations on the parallel computer (100). Eachoperational group (132) of compute nodes is the set of compute nodesupon which a collective parallel operation executes. Each compute nodein the operational group (132) is assigned a unique rank that identifiesthe particular compute node in the operational group (132). Collectiveoperations are implemented with data communications among the computenodes of an operational group. Collective operations are those functionsthat involve all the compute nodes of an operational group (132). Acollective operation is an operation, a message-passing computer programinstruction that is executed simultaneously, that is, at approximatelythe same time, by all the compute nodes in an operational group (132) ofcompute nodes. Such an operational group (132) may include all thecompute nodes (102) in a parallel computer (100) or a subset all thecompute nodes (102). Collective operations are often built aroundpoint-to-point operations. A collective operation requires that allprocesses on all compute nodes within an operational group (132) callthe same collective operation with matching arguments. A ‘broadcast’ isan example of a collective operation for moving data among compute nodesof an operational group. A ‘reduce’ operation is an example of acollective operation that executes arithmetic or logical functions ondata distributed among the compute nodes of an operational group (132).An operational group (132) may be implemented as, for example, an MPI‘communicator.’

‘MPI’ refers to ‘Message Passing Interface,’ a prior art parallelcommunications library, a module of computer program instructions fordata communications on parallel computers. Examples of prior-artparallel communications libraries that may be improved for use insystems configured according to embodiments of the present inventioninclude MPI and the ‘Parallel Virtual Machine’ (‘PVM’) library. PVM wasdeveloped by the University of Tennessee, The Oak Ridge NationalLaboratory and Emory University. MPI is promulgated by the MPI Forum, anopen group with representatives from many organizations that define andmaintain the MPI standard. MPI at the time of this writing is a de factostandard for communication among compute nodes running a parallelprogram on a distributed memory parallel computer. This specificationsometimes uses MPI terminology for ease of explanation, although the useof MPI as such is not a requirement or limitation of the presentinvention.

Some collective operations have a single originating or receivingprocess running on a particular compute node in an operational group(132). For example, in a ‘broadcast’ collective operation, the processon the compute node that distributes the data to all the other computenodes is an originating process. In a ‘gather’ operation, for example,the process on the compute node that received all the data from theother compute nodes is a receiving process. The compute node on whichsuch an originating or receiving process runs is referred to as alogical root.

Most collective operations are variations or combinations of four basicoperations: broadcast, gather, scatter, and reduce. The interfaces forthese collective operations are defined in the MPI standards promulgatedby the MPI Forum. Algorithms for executing collective operations,however, are not defined in the MPI standards. In a broadcast operation,all processes specify the same root process, whose buffer contents willbe sent. Processes other than the root specify receive buffers. Afterthe operation, all buffers contain the message from the root process.

A scatter operation, like the broadcast operation, is also a one-to-manycollective operation. In a scatter operation, the logical root dividesdata on the root into segments and distributes a different segment toeach compute node in the operational group (132). In scatter operation,all processes typically specify the same receive count. The sendarguments are only significant to the root process, whose bufferactually contains sendcount*N elements of a given datatype, where N isthe number of processes in the given group of compute nodes. The sendbuffer is divided and dispersed to all processes (including the processon the logical root). Each compute node is assigned a sequentialidentifier termed a ‘rank.’ After the operation, the root has sentsendcount data elements to each process in increasing rank order. Rank 0receives the first sendcount data elements from the send buffer. Rank 1receives the second sendcount data elements from the send buffer, and soon.

A gather operation is a many-to-one collective operation that is acomplete reverse of the description of the scatter operation. That is, agather is a many-to-one collective operation in which elements of adatatype are gathered from the ranked compute nodes into a receivebuffer in a root node.

A reduction operation is also a many-to-one collective operation thatincludes an arithmetic or logical function performed on two dataelements. All processes specify the same ‘count’ and the same arithmeticor logical function. After the reduction, all processes have sent countdata elements from compute node send buffers to the root process. In areduction operation, data elements from corresponding send bufferlocations are combined pair-wise by arithmetic or logical operations toyield a single corresponding element in the root process' receivebuffer. Application specific reduction operations can be defined atruntime. Parallel communications libraries may support predefinedoperations. MPI, for example, provides the following predefinedreduction operations:

-   -   MPI_MAX maximum    -   MPI_MIN minimum    -   MPI_SUM sum    -   MPI_PROD product    -   MPI_LAND logical and    -   MPI_BAND bitwise and    -   MPI_LOR logical or    -   MPI_BOR bitwise or    -   MPI_LXOR logical exclusive or    -   MPI_BXOR bitwise exclusive or

In addition to compute nodes, the parallel computer (100) includesinput/output (‘I/O’) nodes (110, 114) coupled to compute nodes (102)through the global combining network (106). The compute nodes (102) inthe parallel computer (100) may be partitioned into processing sets suchthat each compute node in a processing set is connected for datacommunications to the same I/O node. Each processing set, therefore, iscomposed of one I/O node and a subset of compute nodes (102). The ratiobetween the number of compute nodes to the number of I/O nodes in theentire system typically depends on the hardware configuration for theparallel computer (102). For example, in some configurations, eachprocessing set may be composed of eight compute nodes and one I/O node.In some other configurations, each processing set may be composed ofsixty-four compute nodes and one I/O node. Such example are forexplanation only, however, and not for limitation. Each I/O nodeprovides I/O services between compute nodes (102) of its processing setand a set of I/O devices. In the example of FIG. 1, the I/O nodes (110,114) are connected for data communications I/O devices (118, 120, 122)through local area network (‘LAN’) (130) implemented using high-speedEthernet.

The parallel computer (100) of FIG. 1 also includes a service node (116)coupled to the compute nodes through one of the networks (104). Servicenode (116) provides services common to pluralities of compute nodes,administering the configuration of compute nodes, loading programs intothe compute nodes, starting program execution on the compute nodes,retrieving results of program operations on the compute nodes, and soon. Service node (116) runs a service application (124) and communicateswith users (128) through a service application interface (126) that runson computer terminal (122).

The parallel computer (100) of FIG. 1 operates generally fordistributing an executable job load file to compute nodes in a parallelcomputer in accordance with embodiments of the present invention. Asmentioned above, some parallel application may be split into parallelprocesses or parallel tasks. For simplicity in this specification asingle compute node is often described as executing a single task.Readers will understand however that a compute node may execute in anynumber of tasks. In the example of FIG. 1, each compute node (102) mayexecute a number of tasks where at least one of the compute nodesexecutes a number of tasks that is different than the number of tasksexecuted by another one of the compute nodes.

Distributing an executable job load file to compute nodes according toembodiments of the present invention is generally implemented on aparallel computer that includes a plurality of compute nodes organizedfor collective operations through at least one data communicationsnetwork. In fact, such computers may include thousands of such computenodes. Each compute node is in turn itself a kind of computer composedof one or more computer processing cores, its own computer memory, andits own input/output adapters. For further explanation, therefore, FIG.2 sets forth a block diagram of an example compute node (102) useful ina parallel computer capable of distributing an executable job load fileto compute nodes according to embodiments of the present invention. Thecompute node (102) of FIG. 2 includes a plurality of processing cores(165) as well as RAM (156). The processing cores (165) of FIG. 2 may beconfigured on one or more integrated circuit dies. Processing cores(165) are connected to RAM (156) through a high-speed memory bus (155)and through a bus adapter (194) and an extension bus (168) to othercomponents of the compute node. Stored in RAM (156) is an applicationprogram (159), a module of computer program instructions that carriesout parallel, user-level data processing using parallel algorithms.

Also stored RAM (156) is a parallel communications library (161), alibrary of computer program instructions that carry out parallelcommunications among compute nodes, including point-to-point operationsas well as collective operations.

A library of parallel communications routines may be developed fromscratch for use in systems according to embodiments of the presentinvention, using a traditional programming language such as the Cprogramming language, and using traditional programming methods to writeparallel communications routines that send and receive data among nodeson two independent data communications networks. Alternatively, existingprior art libraries may be improved to operate according to embodimentsof the present invention. Examples of prior-art parallel communicationslibraries include the ‘Message Passing Interface’ (‘MPI’) library andthe ‘Parallel Virtual Machine’ (‘PVM’) library.

Also stored in RAM (156) is an operating system (162), a module ofcomputer program instructions and routines for an application program'saccess to other resources of the compute node. It is typical for anapplication program and parallel communications library in a computenode of a parallel computer to run a single thread of execution with nouser login and no security issues because the thread is entitled tocomplete access to all resources of the node. The quantity andcomplexity of tasks to be performed by an operating system on a computenode in a parallel computer therefore are smaller and less complex thanthose of an operating system on a serial computer with many threadsrunning simultaneously. In addition, there is no video I/O on thecompute node (102) of FIG. 2, another factor that decreases the demandson the operating system. The operating system (162) may therefore bequite lightweight by comparison with operating systems of generalpurpose computers, a pared down version as it were, or an operatingsystem developed specifically for operations on a particular parallelcomputer. Operating systems that may usefully be improved, simplified,for use in a compute node include UNIX™, Linux™, Windows XP™, AIX™,IBM's i5/OS™, and others as will occur to those of skill in the art.

The example compute node (102) of FIG. 2 includes several communicationsadapters (172, 176, 180, 188) for implementing data communications withother nodes of a parallel computer. Such data communications may becarried out serially through RS-232 connections, through external busessuch as USB, through data communications networks such as IP networks,and in other ways as will occur to those of skill in the art.Communications adapters implement the hardware level of datacommunications through which one computer sends data communications toanother computer, directly or through a network. Examples ofcommunications adapters useful in apparatus useful for distributing anexecutable job load file to compute nodes in a parallel computeraccording to embodiments of the present invention include modems forwired communications, Ethernet (IEEE 802.3) adapters for wired networkcommunications, and 802.11b adapters for wireless networkcommunications.

The data communications adapters in the example of FIG. 2 include aGigabit Ethernet adapter (172) that couples example compute node (102)for data communications to a Gigabit Ethernet (174). Gigabit Ethernet isa network transmission standard, defined in the IEEE 802.3 standard,that provides a data rate of 1 billion bits per second (one gigabit).Gigabit Ethernet is a variant of Ethernet that operates over multimodefiber optic cable, single mode fiber optic cable, or unshielded twistedpair.

The data communications adapters in the example of FIG. 2 include a JTAGSlave circuit (176) that couples example compute node (102) for datacommunications to a JTAG Master circuit (178). JTAG is the usual nameused for the IEEE 1149.1 standard entitled Standard Test Access Port andBoundary-Scan Architecture for test access ports used for testingprinted circuit boards using boundary scan. JTAG is so widely adaptedthat, at this time, boundary scan is more or less synonymous with JTAG.JTAG is used not only for printed circuit boards, but also forconducting boundary scans of integrated circuits, and is also useful asa mechanism for debugging embedded systems, providing a convenientalternative access point into the system. The example compute node ofFIG. 2 may be all three of these: It typically includes one or moreintegrated circuits installed on a printed circuit board and may beimplemented as an embedded system having its own processing core, itsown memory, and its own I/O capability. JTAG boundary scans through JTAGSlave (176) may efficiently configure processing core registers andmemory in compute node (102) for use in dynamically reassigning aconnected node to a block of compute nodes useful in systems fordistributing an executable job load file to compute nodes in a parallelcomputer according to embodiments of the present invention.

The data communications adapters in the example of FIG. 2 include aPoint-To-Point Network Adapter (180) that couples example compute node(102) for data communications to a network (108) that is optimal forpoint-to-point message passing operations such as, for example, anetwork configured as a three-dimensional torus or mesh. ThePoint-To-Point Adapter (180) provides data communications in sixdirections on three communications axes, x, y, and z, through sixbidirectional links: +x (181), −x (182), +y (183), −y (184), +z (185),and −z (186).

The data communications adapters in the example of FIG. 2 include aGlobal Combining Network Adapter (188) that couples example compute node(102) for data communications to a global combining network (106) thatis optimal for collective message passing operations such as, forexample, a network configured as a binary tree. The Global CombiningNetwork Adapter (188) provides data communications through threebidirectional links for each global combining network (106) that theGlobal Combining Network Adapter (188) supports. In the example of FIG.2, the Global Combining Network Adapter (188) provides datacommunications through three bidirectional links for global combiningnetwork (106): two to children nodes (190) and one to a parent node(192).

The example compute node (102) includes multiple arithmetic logic units(‘ALUs’). Each processing core (165) includes an ALU (166), and aseparate ALU (170) is dedicated to the exclusive use of the GlobalCombining Network Adapter (188) for use in performing the arithmetic andlogical functions of reduction operations, including an allreduceoperation. Computer program instructions of a reduction routine in aparallel communications library (161) may latch an instruction for anarithmetic or logical function into an instruction register (169). Whenthe arithmetic or logical function of a reduction operation is a ‘sum’or a ‘logical OR,’ for example, the collective operations adapter (188)may execute the arithmetic or logical operation by use of the ALU (166)in the processing core (165) or, typically much faster, by use of thededicated ALU (170) using data provided by the nodes (190, 192) on theglobal combining network (106) and data provided by processing cores(165) on the compute node (102).

Often when performing arithmetic operations in the global combiningnetwork adapter (188), however, the global combining network adapter(188) only serves to combine data received from the children nodes (190)and pass the result up the network (106) to the parent node (192).Similarly, the global combining network adapter (188) may only serve totransmit data received from the parent node (192) and pass the data downthe network (106) to the children nodes (190). That is, none of theprocessing cores (165) on the compute node (102) contribute data thatalters the output of ALU (170), which is then passed up or down theglobal combining network (106). Because the ALU (170) typically does notoutput any data onto the network (106) until the ALU (170) receivesinput from one of the processing cores (165), a processing core (165)may inject the identity element into the dedicated ALU (170) for theparticular arithmetic operation being perform in the ALU (170) in orderto prevent alteration of the output of the ALU (170). Injecting theidentity element into the ALU, however, often consumes numerousprocessing cycles. To further enhance performance in such cases, theexample compute node (102) includes dedicated hardware (171) forinjecting identity elements into the ALU (170) to reduce the amount ofprocessing core resources required to prevent alteration of the ALUoutput. The dedicated hardware (171) injects an identity element thatcorresponds to the particular arithmetic operation performed by the ALU.For example, when the global combining network adapter (188) performs abitwise OR on the data received from the children nodes (190), dedicatedhardware (171) may inject zeros into the ALU (170) to improveperformance throughout the global combining network (106).

For further explanation, FIG. 3 sets forth a block diagram of an examplePoint-To-Point Adapter (180) useful in systems for distributing anexecutable job load file to compute nodes in a parallel computeraccording to embodiments of the present invention. The Point-To-PointAdapter (180) is designed for use in a data communications networkoptimized for point-to-point operations, a network that organizescompute nodes in a three-dimensional torus or mesh. The Point-To-PointAdapter (180) in the example of FIG. 3 provides data communication alongan x-axis through four unidirectional data communications links, to andfrom the next node in the −x direction (182) and to and from the nextnode in the +x direction (181). The Point-To-Point Adapter (180) of FIG.3 also provides data communication along a y-axis through fourunidirectional data communications links, to and from the next node inthe −y direction (184) and to and from the next node in the +y direction(183). The Point-To-Point Adapter (180) of FIG. 3 also provides datacommunication along a z-axis through four unidirectional datacommunications links, to and from the next node in the −z direction(186) and to and from the next node in the +z direction (185).

For further explanation, FIG. 4 sets forth a block diagram of an exampleGlobal Combining Network Adapter (188) useful in systems fordistributing an executable job load file to compute nodes in a parallelcomputer according to embodiments of the present invention. The GlobalCombining Network Adapter (188) is designed for use in a networkoptimized for collective operations, a network that organizes computenodes of a parallel computer in a binary tree. The Global CombiningNetwork Adapter (188) in the example of FIG. 4 provides datacommunication to and from children nodes of a global combining networkthrough four unidirectional data communications links (190), and alsoprovides data communication to and from a parent node of the globalcombining network through two unidirectional data communications links(192).

For further explanation, FIG. 5 sets forth a line drawing illustratingan example data communications network (108) optimized forpoint-to-point operations useful in systems capable of distributing anexecutable job load file to compute nodes in a parallel computeraccording to embodiments of the present invention. In the example ofFIG. 5, dots represent compute nodes (102) of a parallel computer, andthe dotted lines between the dots represent data communications links(103) between compute nodes. The data communications links areimplemented with point-to-point data communications adapters similar tothe one illustrated for example in FIG. 3, with data communicationslinks on three axis, x, y, and z, and to and fro in six directions +x(181), −x (182), +y (183), −y (184), +z (185), and −z (186). The linksand compute nodes are organized by this data communications networkoptimized for point-to-point operations into a three dimensional mesh(105). The mesh (105) has wrap-around links on each axis that connectthe outermost compute nodes in the mesh (105) on opposite sides of themesh (105). These wrap-around links form a torus (107). Each computenode in the torus has a location in the torus that is uniquely specifiedby a set of x, y, z coordinates. Readers will note that the wrap-aroundlinks in the y and z directions have been omitted for clarity, but areconfigured in a similar manner to the wrap-around link illustrated inthe x direction. For clarity of explanation, the data communicationsnetwork of FIG. 5 is illustrated with only 27 compute nodes, but readerswill recognize that a data communications network optimized forpoint-to-point operations for use in distributing an executable job loadfile to compute nodes in a parallel computer in accordance withembodiments of the present invention may contain only a few computenodes or may contain thousands of compute nodes. For ease ofexplanation, the data communications network of FIG. 5 is illustratedwith only three dimensions, but readers will recognize that a datacommunications network optimized for point-to-point operations for usein distributing an executable job load file to compute nodes in aparallel computer in accordance with embodiments of the presentinvention may in fact be implemented in two dimensions, four dimensions,five dimensions, and so on. Several supercomputers now use fivedimensional mesh or torus networks, including, for example, IBM's BlueGene Q™.

For further explanation, FIG. 6 sets forth a line drawing illustratingan example global combining network (106) useful in systems capable ofdistributing an executable job load file to compute nodes in a parallelcomputer according to embodiments of the present invention. The exampledata communications network of FIG. 6 includes data communications links(103) connected to the compute nodes so as to organize the compute nodesas a tree. In the example of FIG. 6, dots represent compute nodes (102)of a parallel computer, and the dotted lines (103) between the dotsrepresent data communications links between compute nodes. The datacommunications links are implemented with global combining networkadapters similar to the one illustrated for example in FIG. 4, with eachnode typically providing data communications to and from two childrennodes and data communications to and from a parent node, with someexceptions. Nodes in the global combining network (106) may becharacterized as a physical root node (202), branch nodes (204), andleaf nodes (206). The physical root (202) has two children but no parentand is so called because the physical root node (202) is the nodephysically configured at the top of the binary tree. The leaf nodes(206) each has a parent, but leaf nodes have no children. The branchnodes (204) each has both a parent and two children. The links andcompute nodes are thereby organized by this data communications networkoptimized for collective operations into a binary tree (106). Forclarity of explanation, the data communications network of FIG. 6 isillustrated with only 31 compute nodes, but readers will recognize thata global combining network (106) optimized for collective operations foruse in distributing an executable job load file to compute nodes in aparallel computer in accordance with embodiments of the presentinvention may contain only a few compute nodes or may contain thousandsof compute nodes.

In the example of FIG. 6, each node in the tree is assigned a unitidentifier referred to as a ‘rank’ (250). The rank actually identifies atask or process that is executing a parallel operation according toembodiments of the present invention. Using the rank to identify a nodeassumes that only one such task is executing on each node. To the extentthat more than one participating task executes on a single node, therank identifies the task as such rather than the node. A rank uniquelyidentifies a task's location in the tree network for use in bothpoint-to-point and collective operations in the tree network. The ranksin this example are assigned as integers beginning with 0 assigned tothe root tasks or root node (202), 1 assigned to the first node in thesecond layer of the tree, 2 assigned to the second node in the secondlayer of the tree, 3 assigned to the first node in the third layer ofthe tree, 4 assigned to the second node in the third layer of the tree,and so on. For ease of illustration, only the ranks of the first threelayers of the tree are shown here, but all compute nodes in the treenetwork are assigned a unique rank.

For further explanation, FIG. 7 sets forth a flow chart illustrating anexample method for distributing an executable job load file (606) tocompute nodes in a parallel computer (602) according to embodiments ofthe present invention. The parallel computer (602) of FIG. 7 includes aplurality of compute nodes (604, 608, 612, 614, 616) coupled for datacommunications over one or more data communications networks asillustrated above with reference to FIG. 1. In the example method ofFIG. 7, the compute nodes (604, 608, 612, 614, 616) may be coupled fordata communications over data communications network such as an Ethernetnetwork, a global combining network, a JTAG network, a point-to-pointnetwork, or any combination thereof as described above.

In the example method of FIG. 7, the compute nodes (604, 608, 612, 614,616) may be logically organized into a tree structure such as a binarytree. In such an example, one of the compute nodes may be identified asbeing the root compute node (604) representing the logical root of thetree. One of the compute nodes may also identified as being a parentcompute node (608), as that compute node (608) can represent the logicalparent of compute node (612). Likewise, two compute nodes are identifiedas being descendent compute nodes (614, 616), as the descendent computenodes (614, 616) can represent the logical children of compute node(612). As such, the designation of specific nodes as being a ‘parent,’ a‘child,’ or a ‘descendent’ is a relative designation of the logicalrelationship between the designated nodes and compute node (612).

In the example method of FIG. 7, one or more of the compute nodes (604,608, 612, 614, 616) may participate in the execution of a particularjob. In order to participate in the execution of a particular job, acompute node (604, 608, 612, 614, 616) may need to receive an executablejob load file (606) that enables the compute node to load the necessarycode and data needed to participate in the execution of the particularjob. The executable job load file (606) may include, for example,computer program instructions needed to participate in the execution ofthe job, data needed to participate in the execution of the job, and soon. In such a way, a compute node may utilize the executable job loadfile (606) to configure the compute node for participation in theexecution of a job. Readers will appreciate that each compute node thatis participating in the execution of a particular job may therefore needto receive the executable job load file (606) prior to participating inthe execution of the job.

The example method of FIG. 7 includes determining (618), by a computenode (612) in the parallel computer (602), whether the compute node(612) is participating in a job. In the example method of FIG. 7, thejob may represent some computing task that is distributed over aplurality of compute nodes (608, 612, 614, 616) such that each computenode (608, 612, 614, 616) participating in the job carries out someportion of the job. Each compute node (608, 612, 614, 616) can determinewhether it is participating in the execution of a particular job byexamining an identifier associated with the job. In such an example,when a particular compute node (608, 612, 614, 616) is booted, thecompute node (608, 612, 614, 616) may be booted with informationidentifying one or more jobs that the compute node (608, 612, 614, 616)will participate in executing. As such, determining (618) whether thecompute node (612) is participating in the execution of a particular jobmay be carried out by the compute node (612) comparing a list ofidentifiers for jobs that the compute node (612) will participate inexecuting to an identifier for the particular job.

The example method of FIG. 7 also includes determining (620), by thecompute node (612) in the parallel computer (602), whether a descendentcompute node (614, 616) is participating in the job. In the examplemethod of FIG. 7, a descendent compute node (614, 616) may represent achild of the compute node (612), a grandchild of the compute node (612),and so on. As described above, each compute node may determine whetherit is participating in the execution of a particular job by comparing alist of identifiers for jobs that the compute node will participate inexecuting to an identifier for the particular job. Each descendentcompute node (614, 616) may therefore notify its parent that thedescendent compute node (614, 616) will be or will not be participatingin the execution of the particular job. Each descendent compute node(614, 616) may notify its parent that the descendent compute node (614,616) will be or will not be participating in the execution of theparticular job, for example, by sending a message to its parent node, bywriting a value to a flag stored at a predetermined address in sharedmemory that indicates whether a node is participating in the executionof the particular job, and so on. In such a way, determining (620)whether a descendent compute node (614, 616) is participating in the jobmay be carried out by the compute node (612) monitoring a message queue,shared memory, or other communications channel for an indication fromthe descendent compute node (614, 616) that the descendent compute node(614, 616) is or is not participating in the execution of the particularjob.

The example method of FIG. 7 also includes communicating (624), by thecompute node (612) to a parent compute node (608), an identification ofa data communications link over which the compute node (612) receivesdata from the parent compute node (608). In the example method of FIG.7, communicating (624) an identification of a data communications linkover which the compute node (612) receives data from the parent computenode (608) may be carried out by the compute node (612) sending amessage to the parent compute node (608) that includes an identifier ofthe of a data communications link over which the compute node (612)receives data from the parent compute node (608), by the compute node(612) writing a value to a predetermined address in memory that isshared with the parent compute node (608), where the value includes anidentifier of the of a data communications link over which the computenode (612) receives data from the parent compute node (608), and so on.

In the example method of FIG. 7, communicating (624) an identificationof a data communications link over which the compute node (612) receivesdata from the parent compute node (608) may be carried out in responseto affirmatively (617) determining that the compute node (612) isparticipating in the job or in response to affirmatively (619)determining that one or more of the descendent compute nodes (614, 616)is participating in the job. In an example in which the compute node(612) is participating in the job, the compute node (612) will need toreceive the executable job load file (606) to configure the compute node(612) for participation in executing the job. Likewise, in an example inwhich one or more of the descendent compute nodes (614, 616) isparticipating in the job, the compute node (612) will need to receivethe executable job load file (606) for subsequent transmission to theone or more of the descendent compute nodes (614, 616) participating inthe job. As such, the compute node (612) will need to receive theexecutable job load file (606) when the compute node (612) isparticipating in the job or when one or more of the descendent computenodes (614, 616) is participating in the job. By communicating (624) anidentification of a data communications link over which the compute node(612) receives data from the parent compute node (608), the compute node(612) can identify a communications link over which the parent computenode (608) may later transmit the executable job load file (606) to thecompute node (612). In the example method of FIG. 7, steps 618, 620, and624 are carried out iteratively for a predetermined number ofiterations. The iterative nature of steps 618, 620, and 624 will beexplained in greater detail below, specifically with reference to anexample described below and depicted in FIG. 8.

The example method of FIG. 7 also includes constructing (632) a classroute for the job. In the example method of FIG. 7, the class route canidentify all compute nodes participating in the job, as well as the datacommunications links between the compute nodes participating in the job.In such an example, constructing (632) a class route for the job maytherefore be carried out by identifying all compute nodes that areeither participating in the execution of the job or have descendantcompute nodes that are participating in the execution of the job. Insuch an example, all compute nodes that are either participating in theexecution of the job or have descendant compute nodes that areparticipating in the execution of the job may therefore be part of asub-network of compute nodes in the parallel computer that comprise themembers of the class route. A class route identifier may be associatedwith each distinct class route such that messages that are broadcast bythe root node can include a class route identifier in a message header,identifying which compute nodes are to receive and process the message.Each compute node in the parallel computer may include a routing tablethat associates a class route identifier with one or more egress portson the compute node for forwarding a message received by the computenode that includes the class route identifier, such that the message isonly broadcast to the members of the corresponding class route.

The example method of FIG. 7 also includes broadcasting (634) theexecutable load file (606) for the job along the class route for thejob. In the example method of FIG. 7, broadcasting (634) the executableload file (606) for the job along the class route for the job may becarried out by the root compute node (604) generating a broadcastmessage that includes the executable load file (606) as message payloadand also includes a class route identifier in the message header. Insuch an example, the root compute node (604) may send the broadcastmessage to its children, which will subsequently determine which egressport through which to forward the message using the class routeidentifier and an internal routing table. As this process is repeated,the broadcast message will be passed along to each of the compute nodesthat are members of the class route associated with the class routeidentifier.

For further explanation, FIG. 8 sets forth an example of a parallelcomputer where the compute nodes in the parallel computer are logicallyorganized into a binary tree. The example depicted in FIG. 8specifically identifies five compute nodes in the parallel computer. Onecompute node of the parallel computer that is specifically identified inFIG. 8 is the root compute node (704), which represents the logical rootof the binary tree. The remaining compute nodes of the parallel computerthat are specifically identified are participating compute nodes (704,710) and non-participating compute nodes (706, 708). In the exampledepicted in FIG. 8, participating compute nodes (704, 710) representcompute nodes that are participating in the execution of a job. Thenon-participating compute nodes (706, 708) depicted in FIG. 8 representcompute nodes that are not participating in the execution of the job. Asmentioned above, steps 618, 620, and 624 of FIG. 7 are carried outiteratively for a predetermined number of iterations. The iterativenature of steps 618, 620, and 624 of FIG. 7 will be explained withreference to the example depicted in FIG. 8.

In a first iteration of steps 618, 620, and 624 of FIG. 7, each of thenon-root compute node (704, 706, 708, 710) determines (618 of FIG. 7)whether it is participating in the execution of a job. In the exampledepicted in FIG. 8, the participating compute nodes (704, 710) willdetermine that they are participating in the execution of the job andthe non-participating compute nodes (706, 708) will determine that theyare not participating in the execution of the job. Each of the non-rootcompute nodes (704, 706, 708, 710) will subsequently determine (620 ofFIG. 7) whether any of its descendant compute nodes are participating inthe execution of the job. During the first iteration, however, none ofthe compute nodes (704, 706, 708, 710) will have received a notificationthat one of their descendant compute nodes are participating in theexecution of the job. Each of the non-root compute nodes (704, 706, 708,710) will then communicate (624 of FIG. 7), to its parent compute node,an identification of a data communications link over which the computenode receives data from the parent compute node in response toaffirmatively determining that the compute node is participating in thejob or in response to affirmatively determining that one or more of thechild nodes is participating in the job. In the example depicted in FIG.8, participating compute node (704) will therefore communicate anidentification of a data communications link over which participatingcompute node (704) receives data from its parent compute node, which isthe root compute node (702). Likewise, participating compute node (710)will communicate an identification of a data communications link overwhich participating compute node (710) receives data from its parentcompute node, which is the non-participating compute node (708). In suchan example, the non-participating compute nodes (706, 708) willcommunicate an identification of a data communications link to a parentcompute node as neither node is participating in the execution of thejob nor is either compute node aware that a descendant compute node isparticipating in the execution of the job.

In a second iteration of steps 618, 620, and 624 of FIG. 7, each of thenon-root compute nodes (704, 706, 708, 710) determines (618 of FIG. 7)whether it is participating in the execution of a job as describedabove. Each of the non-root compute nodes (704, 706, 708, 710) willsubsequently determine (620 of FIG. 7) whether any of its descendantcompute nodes are participating in the execution of the job. During thesecond iteration, non-participating compute node (708) will havereceived a notification that one of its descendant compute nodes,participating computing node (710), is participating in the execution ofthe job. Each of the non-root compute nodes (704, 706, 708, 710) willthen communicate (624 of FIG. 7), to its parent compute node, anidentification of a data communications link over which the compute nodereceives data from the parent compute node if the compute node isparticipating in the job or if one or more of the descendant nodes isparticipating in the job. In the example depicted in FIG. 8, theparticipating compute nodes (704, 710) will again communicate anidentification of a data communications link over which participatingcompute nodes (704, 710) receive data from its parent compute node. Inaddition, non-participating compute node (708) will also communicate anidentification of a data communications link over which thenon-participating compute node (708) receives data from its parentcompute node, which is non-participating compute node (706). Thenon-participating compute node (708) will communicate an identificationof a data communications link over which the non-participating computenode (708) receives data from non-participating compute node (706) byvirtue of non-participating compute node (708) determining that one ofits descendant compute nodes is participating in the execution of thejob.

In a third iteration of steps 618, 620, and 624 of FIG. 7, each of thenon-root compute nodes (704, 706, 708, 710) determines (618 of FIG. 7)whether it is participating in the execution of a job as describedabove. Each of the non-root compute nodes (704, 706, 708, 710) willsubsequently determine (620 of FIG. 7) whether any of its descendantcompute nodes are participating in the execution of the job. During thethird iteration, non-participating compute node (706) will have receiveda notification that one of its descendent compute nodes, participatingcomputing node (710), is participating in the execution of the job. Eachof the non-root compute nodes (704, 706, 708, 710) will then communicate(624 of FIG. 7), to its parent compute node, an identification of a datacommunications link over which the compute node receives data from theparent compute node if the compute node is participating in the job orif one or more of the descendant nodes is participating in the job. Inthe example depicted in FIG. 8, the compute nodes (704, 708, 710) willagain communicate an identification of a data communications link overwhich the compute nodes (704, 708, 710) receive data from its parentcompute node. In addition, non-participating compute node (706) willalso communicate an identification of a data communications link overwhich the non-participating compute node (706) receives data from itsparent compute node, which is participating compute node (704). Thenon-participating compute node (706) will communicate an identificationof a data communications link over which the non-participating computenode (706) receives data from participating compute node (704) by virtueof non-participating compute node (706) determining that one of itsdescendent compute nodes is participating in the execution of the job.

Upon completion of the steps described above, the method depicted inFIG. 7 would continue by constructing (632 of FIG. 7) a class route forthe job that includes the data communications link between root computenode (702) and participating compute node (704), the data communicationslink between participating compute node (704) and non-participatingcompute node (706), the data communications link betweennon-participating compute node (706) and non-participating compute node(708), and the data communications link between non-participatingcompute node (708) and participating compute node (710). The method ofFIG. 7 would continue by subsequently broadcasting (634 of FIG. 7) theexecutable load file (606 of FIG. 7) for the job along the class routefor the job.

Readers will appreciate that in this example, after three iterations,each compute node that has a descendant that is participating in theexecution of the job will have been notified that it has a descendantthat is participating in the execution of the job. In an example inwhich only leaf compute nodes in a binary tree were participating in theexecution of the job, filtering a notification to each compute node inthe path of each participating leaf compute node would require a numberof iterations that is equal to the number of hops between aparticipating leaf compute node and a root compute node. That is,filtering a notification to each compute node in the path of eachparticipating leaf compute node would require a number of iterationsthat is equal to the number of compute nodes that are ancestors of aparticipating leaf compute node. In such a way, the predetermined numberof iterations may be determined in dependence upon a depth of the binarytree.

For further explanation, FIG. 9 sets forth a flow chart illustrating anadditional example method for distributing an executable job load fileto compute nodes in a parallel computer according to embodiments of thepresent invention. The example method of FIG. 9 is similar to theexample method of FIG. 7 as it also includes determining (618) whetherthe compute node (612) is participating in the job, determining (620)whether a descendent compute node (614, 616) is participating in thejob, communicating (624) an identification of a data communications linkover which the compute node (612) receives data from the parent computenode (608), constructing (632) a class route for the job, andbroadcasting (634) the executable load file (606) for the job along theclass route for the job.

In the example method of FIG. 9, each compute node (604, 608, 612, 614,616) in the parallel computer (602) may maintain a participation vector.Such a participation vector may be embodied as a data structure used toidentify which compute nodes that are connected to the compute node(612) maintaining the participation vector are participating in theexecution of a job. In such an example, each compute node (604, 608,612, 614, 616) may maintain its own participation vector, where eachentry in the participation vector is associated with a communicationslink between the compute node (604, 608, 612, 614, 616) that maintainsthe participation vector and another compute node (604, 608, 612, 614,616). In such an example, each value in the participation vector mayindicate whether the compute node (604, 608, 612, 614, 616) on the otherend of the communications link is a compute node (604, 608, 612, 614,616) participating in the execution of the job.

In the example method of FIG. 9, communicating (624) an identificationof a data communications link over which the compute node (612) receivesdata from the parent compute node (608) can include performing (802), bythe compute node (612), an atomic direct put operation. In the examplemethod of FIG. 9, a direct put operation may be an operation that writesdata to a specific location in memory. The direct put operation of FIG.9 is atomic in the sense that the operation completes in a unit of timethat is indivisible, such as a single processor cycle, such that anatomic operation is performed entirely or not performed at all.

In the example method of FIG. 9, performing (802) an atomic direct putoperation may be carried out by the compute node (612) writing a valueto a location in memory of the parent compute (608). For example, thecompute node (612) may write a value indicating that the compute node(612) or one of its descendant compute nodes (614, 616) is participatingin the execution of a job to a location in memory of the parent compute(608), such as a location in memory where the parent compute node (608)stores its participation vector. In such a way, the compute node (612)may update the participation vector of its parent compute node (608) toindicate that the compute node (612) or one of its descendant computenodes (614, 616) is participating in the execution of the job.

In the example method of FIG. 9, communicating (624) an identificationof a data communications link over which the compute node (612) receivesdata from the parent compute node (608) can also include performing(804) a barrier operation after the atomic direct put operation. In theexample method of FIG. 9, a barrier operation represents a type ofsynchronization amongst a group of processes in which each process muststop executing upon reaching a particular point. Only when all processesreach the same point of execution and stop is the barrier operationcomplete and the processes may continue executing. Barrier operations ina parallel computer that includes many compute nodes, each of which maybe executing a number of processes, may be carried out locally, withtasks on a single compute node and globally amongst many tasks of manycompute nodes.

In the example method of FIG. 9, each of the compute nodes that areperforming the steps depicted in FIG. 9 may perform (804) a barrieroperation as part of communicating (624) an identification of a datacommunications link over which the compute node (612) receives data fromthe parent compute node (608). In such a way, each of the compute nodesthat are performing the steps depicted in FIG. 9 can complete the stepof communicating (624) an identification of a data communications linkover which the compute node (612) receives data from the parent computenode (608) prior to moving execution flow to step 626.

For further explanation, FIG. 10 sets forth a flow chart illustrating anadditional example method for distributing an executable job load fileto compute nodes in a parallel computer according to embodiments of thepresent invention. The example method of FIG. 10 is similar to theexample method of FIG. 7 as it also includes determining (618) whetherthe compute node (612) is participating in the job, determining (620)whether a descendent compute node (614, 616) is participating in thejob, communicating (624), to a parent compute node (608), anidentification of a data communications link over which the compute node(612) receives data from the parent compute node (608), constructing(632) a class route for the job, and broadcasting (634) the executableload file (606) for the job along the class route for the job.

The example method of FIG. 10 also includes maintaining (902), by eachcompute node in the parallel computer, a participation vector. Asdescribed above, the participation vector may be embodied as a datastructure used to identify which compute nodes that are connected to thecompute node (612) maintaining the participation vector areparticipating in the execution of a job. In such an example, eachcompute node (604, 608, 612, 614, 616) may maintain its ownparticipation vector, where each entry in the participation vector isassociated with a communications link between the compute node (604,608, 612, 614, 616) that maintains the participation vector and anothercompute node (604, 608, 612, 614, 616). In such an example, each valuein the participation vector may indicate whether the compute node (604,608, 612, 614, 616) on the other end of the communications link is acompute node (604, 608, 612, 614, 616) participating in the execution ofthe job. In the example method of FIG. 10, maintaining (902) aparticipation vector may be carried out by each compute node (604, 608,612, 614, 616) in the parallel computer (602) storing a data structure,for example, in memory of the particular compute node (604, 608, 612,614, 616) that is maintaining (902) the participation vector or inmemory that is otherwise accessible to the particular compute node (604,608, 612, 614, 616) that is maintaining (902) the participation vector.

Consider an example in which a particular compute node (612) is coupledfor data communications with five other compute nodes via apoint-to-point adapter that is similar to the point-to-point adapterdepicted in FIG. 3. In such an example, assume that compute node (612)is coupled for data communications with each of the other compute nodesvia an ingress link for incoming communications and a separate egresslink for outgoing communications. The participation vector maintained byeach compute node may therefore include ten bits—one bit for the ingresslink and one bit for the egress link that couples the compute node toeach of the five other compute nodes. In such an example, a value of ‘1’in a particular bit location within the participation vector mayindicate that the compute node on the other end of the communicationslinks associated with the bit location in the participation vector isparticipating in the execution of the job—and that the communicationslink associated with the bit location in the participation vector is thecommunications link over which the participating compute node receivesdata from its logical parent. In view of the fact that the compute nodesin the example of FIG. 10 are logically organized into a binary tree,one of the five other compute nodes that the particular compute node(612) is coupled for data communications with represents a parentcompute node (608). Furthermore, at most two of the five other computenodes that the particular compute node (612) is coupled for datacommunications with represent child compute nodes, which arecharacterized here as descendent compute nodes (614, 616).

In the example method of FIG. 10, maintaining (902) a participationvector can include performing (904) an atomic OR operation. In theexample method of FIG. 10, a subject compute node will be part of theclass route for a job if the subject compute node is participating inthe execution of the job or if a descendent compute node of the subjectcompute node is participating in the execution of the job. When adescendent compute node of the subject compute node is participating inthe execution of the job, the descendent compute node may send a bitpattern to the subject compute node indicating which data communicationslink of the subject application should utilize for sending a load fileto the descendent compute node. Because the subject compute node caninclude more than one descendent compute node, the subject compute nodemay receive such a bit pattern from multiple descendent compute nodes.

Consider an example in which the subject compute node includes apoint-to-point adapter as depicted in FIG. 3. In such an example, assumethat the subject compute node therefore maintains (902) a 10 bitparticipation vector, where each of the 10 data communications linksover which the point-to-point adapter may facilitate data communicationsis associated with a particular bit in the 10 bit participation vector.In such an example, a first descendent compute node may send a bitpattern of ‘0100000000’ to the subject compute node, where the value of‘1’ in the second bit indicates that the subject compute node shouldsend a load file to the first descendent compute node over a datacommunications link associated with the second position of the bitpattern. A second compute node, however, may send a bit pattern of‘0001000000’ to the subject compute node, where the value of ‘1’ in thefourth bit indicates that the subject compute node should send a loadfile to the second descendent compute node over a data communicationslink associated with the fourth position of the bit pattern. In such away, maintaining (902) a participation vector can include performing(904) an atomic OR operation on the bit patterns received from eachdescendent compute node, resulting in a participation vector of‘0101000000.’ In such an example, the values of ‘1’ in the second andfourth bit positions can indicate that the subject compute node shouldbroadcast the load file over the communications links associated withthe second and fourth bit positions of the participation vector.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

It will be understood from the foregoing description that modificationsand changes may be made in various embodiments of the present inventionwithout departing from its true spirit. The descriptions in thisspecification are for purposes of illustration only and are not to beconstrued in a limiting sense. The scope of the present invention islimited only by the language of the following claims.

What is claimed is:
 1. An apparatus for distributing an executable jobload file to compute nodes in a parallel computer, the parallel computercomprising a plurality of compute nodes coupled for data communicationsover a data communications network, the apparatus comprising a computerprocessor, a computer memory operatively coupled to the computerprocessor, the computer memory having disposed within it computerprogram instructions that, when executed by the computer processor,cause the apparatus to carry out the steps of: iteratively for apredetermined number of iterations: determining, by a compute node inthe parallel computer, whether the compute node is participating in ajob; determining, by the compute node in the parallel computer, whethera descendant compute node is participating in the job; responsive todetermining that the compute node is participating in the job or thatthe descendant compute node is participating in the job, communicating,by the compute node to a parent compute node, an identification of adata communications link over which the compute node receives data fromthe parent compute node; constructing a class route for the job, whereinthe class route identifies all compute nodes participating in the joband all data communications links between each of the compute nodesparticipating in the job, wherein each compute node in the parallelcomputer includes a routing table that associates a class routeidentifier with one or more egress ports on the compute node forforwarding a message received by the compute node that includes theclass route identifier; and broadcasting the executable load file forthe job along the class route for the job, wherein the executable loadfile is included in a message that also includes a class routeidentifier for the job, including identifying, by each compute nodeparticipating in the job, from the routing table in the compute node, anegress port within the compute node to utilize when forwarding themessage.
 2. The apparatus of claim 1 wherein the plurality of computenodes in the parallel computer are organized into a binary tree and thepredetermined number of iterations is determined in dependence upon adepth of the binary tree.
 3. The apparatus of claim 1 whereincommunicating, by the compute node to the parent compute node, theidentification of the data communications link over which the computenode receives data from the parent compute node further comprisesperforming, by the compute node, an atomic direct put operation.
 4. Theapparatus of claim 3 further comprising computer program instructionsthat, when executed by the computer processor, cause the apparatus tocarry out the step of performing a barrier operation after the atomicdirect put operation.
 5. The apparatus of claim 1 further comprisingcomputer program instructions that, when executed by the computerprocessor, cause the apparatus to carry out the step of maintaining, byeach compute node in the parallel computer, a participation vector,wherein each entry in the participation vector is associated with acommunications link between the compute node and another compute node.6. The apparatus of claim 5 wherein maintaining, by each compute node inthe parallel computer, a participation vector further comprisesperforming an atomic OR operation.
 7. A computer program product fordistributing an executable job load file to compute nodes in a parallelcomputer, the parallel computer comprising a plurality of compute nodescoupled for data communications over a data communications network, thecomputer program product disposed upon a non-transitory computerreadable medium, the computer program product comprising computerprogram instructions that, when executed, cause a computer to carry outthe steps of: iteratively for a predetermined number of iterations:determining, by a compute node in the parallel computer, whether thecompute node is participating in a job; determining, by the compute nodein the parallel computer, whether a descendant compute node isparticipating in the job; responsive to determining that the computenode is participating in the job or that the descendant compute node isparticipating in the job, communicating, by the compute node to a parentcompute node, an identification of a data communications link over whichthe compute node receives data from the parent compute node;constructing a class route for the job, wherein the class routeidentifies all compute nodes participating in the job and all datacommunications links between each of the compute nodes participating inthe job, wherein each compute node in the parallel computer includes arouting table that associates a class route identifier with one or moreegress ports on the compute node for forwarding a message received bythe compute node that includes the class route identifier; andbroadcasting the executable load file for the job along the class routefor the job, wherein the executable load file is included in a messagethat also includes a class route identifier for the job, includingidentifying, by each compute node participating in the job, from therouting table in the compute node, an egress port within the computenode to utilize when forwarding the message.
 8. The computer programproduct of claim 7 wherein the plurality of compute nodes in theparallel computer are organized into a binary tree and the predeterminednumber of iterations is determined in dependence upon a depth of thebinary tree.
 9. The computer program product of claim 7 whereincommunicating, by the compute node to the parent compute node, theidentification of the data communications link over which the computenode receives data from the parent compute node further comprisesperforming, by the compute node, an atomic direct put operation.
 10. Thecomputer program product of claim 9 further comprising computer programinstructions that, when executed, cause the computer to carry out thestep of performing a barrier operation after the atomic direct putoperation.
 11. The computer program product of claim 7 furthercomprising computer program instructions that, when executed, cause thecomputer to carry out the step of maintaining, by each compute node inthe parallel computer, a participation vector, wherein each entry in theparticipation vector is associated with a communications link betweenthe compute node and another compute node.
 12. The computer programproduct of claim 7 wherein maintaining, by each compute node in theparallel computer, a participation vector further comprises performingan atomic OR operation.
 13. The computer program product of claim 7wherein the computer readable medium comprises a storage medium.